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Abstract 

Imparting real world experiences in a software verification and validation (SV&V) course is often a challenge 
due to the lack of effective active learning tools. This pedagogical requirement is important because graduates are 
expected to develop software that meets rigorous quality standards in functional and application domains. 
Realizing the necessity the authors designed and developed 42 delivery hours of active learning tools consisting of 
Case Studies, Class Exercises, and Case Study Videos for use in courses that impart knowledge on SV&V topics 
viz. requirements engineering, software reviews, configuration management, and software testing. Four key skill 
areas sought after by employers, namely communication skills, applied knowledge of methods, applied 
knowledge of tools, and research exposure are used to drive the development funded by a National Science 
Foundation grant and perfected through an industry-academia partnership. 

In this paper, we discuss in detail the four project plans the researchers and their industry counterparts followed 
over the past two years in the development and eventual dissemination of the active learning tools. A course 
enhancement plan was used to drive activities related to reviewing, enhancing, and modularizing modules, 
identified by a gap analysis performed by focus groups comprised of industry and academic partners. The course 
delivery plan was used to drive activities related to developing content delivery strategies. An evaluation and 
assessment plan was used to drive activities related to periodically evaluating student learning and assessing the 
project. And finally a course dissemination plan is being used to drive activities related to distributing course 
modules and assessment reports. The tools have been shared through two workshops and other means with 
instructors in universities and industry partners. 

Keywords: active learning tools, case studies, configuration management, requirements engineering, reviews, 
Software Verification & Validation, testing 

1. Introduction 

In 2015, major software glitches affected numerous companies: Bloomberg (inoperable trading terminals), Royal 
Bank of Scotland (payments failure), Nissan (airbag sensory detectors malfunction), Starbucks (register 
malfunction) and the F 35 Joint Strike Fighter (targets detection failure). Each of these resulted in financial, 
brand, and more important bodily damages (Jee, 2016). Software is now ubiquitous and software has also 
become critical. With mission critical and high-risk applications that have human lives and resources dependent 
on software applications, it is imperative to not only test for, but aim for zero defects. The irony is that even after 
decades of development, the software industry continues to spend considerable time and resources dealing with 
the quality problem. In the US alone in 2007, the cost of failed software was estimated to be upwards of $75 
billion in re-work costs and abandoned systems (Michaels, 2014). 

The fundamental challenge towards a solution that will improve software quality lies in the people and processes 
that develop and produce software. Acharya, Manohar, Schilling, Ansari, and Wu (2014) reason that firstly, there 
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is not enough awareness of the Software Verification & Validation (SV&V) benefits, and secondly, there are a 
lack of practitioners who understand the SV&V topics and processes adequately. Both the lack of awareness and 
personnel shortage considerably hinder significant progress in software project success rates. Furthermore, 
Acharya et al. (2014) argue the root cause to be the lack of up-to-date SV&V courseware. To address this situation, 
a SV&V course curriculum has been improved at the author’s institution through a project funded by a National 
Science Foundation—Transforming Undergraduate Education in Science, Technology, Engineering, and 
Mathematics (NSF-TUES) grant. 

The goal of this project is to enhance and transform a SV&V course by incorporating academic research and 
industry best practices through an academia-industry partnership. This project achieved the following objectives: 

1) Critically examined the existing SV&V course contents, 2) Identified areas where improvements could be 
made in pedagogy, 3) Developed 42 delivery hours of active learning tools, 4) Developed course delivery 
strategies, 5) Integrated and delivered new pedagogical tools in the course, 6) Performed assessments and 
evaluations of the effectiveness of these tools, and 7) Disseminated course modules and assessment reports. At 
this time dissemination to more institutions is being carried out. 

This project, scheduled for completion in August 2017, targets both undergraduate students and software 
practitioners and: 

1) Improves SV&V knowledge and skills of students & practitioners, 

2) Helps evolve a SV&V community, and 

3) Improves SV&V teaching and learning opportunities. 

Certain contents developed through the project can also be easily adopted to teaching SV&V in graduate 
programs. The end result is expected to improve product and process quality levels in the software development 
community, resulting in a larger and more skilled SV&V user community. This paper introduces this project, 
describes the academia-industry partnership, and provides an in-depth description of the project’s execution and 
outcomes. 

2. Project Partners 

Academia-industry partnership is the key to achievement of the project outcomes. From the conceptual stage, it 
was believed that by working hand in hand with the industry, both the academia and industry would benefit. 
Students would gain knowledge that is currently practiced in the industry and which they would likely use in 
their professional career, and industry would be able to train/retrain their personnel using a tested delivery 
strategy. 

2.1 Industry’ Partners—Formation and Role 

Understanding project requirements, studying possible partnerships, and selling the project vision was 
instrumental in getting the industry on-board. Three industry partners—Eaton Electrical Corporation, 
ServiceLink, and JDA Software Group—have supported this project from the time it was conceived. Additional 
partners, PNC Bank and ANSYS, were invited as this project progressed and the need for diversification was 
realized. These companies are either large software companies or companies with large software development 
activities in the areas of electrical meters, mortgage, and intelligent pricing and revenue management. The 
project team is open to more industry joining the partnership but is selective in the industry domain. The role of 
the industry partner is well defined and involves the following: 

• Critically reviewing and identifying knowledge gaps in SV&V courseware 

• Assisting in developing course modules 

• Delivering expert lecture sessions to undergraduate students at partner universities when requested 

• Delivering training programs to industry practitioners 

• Assessing student learning 

2.2 Academic Partners—Formation and Role 

The author’s institute is collaborating with two categories of academic partners: Development Academic 
Partners and Implementation Academic Partners. These academic partners offer one or more bachelor degrees in 
the following areas: Software Engineering, Computer Science, Computer Engineering, and Electrical 
Engineering. These partners also share a strong desire to strengthen their programs. 
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2.3 Development Academic Partner Activities 

Distinguished faculty members from the Milwaukee School of Engineering and Virginia State University (a 
HBCU partner) have supported this project from the very beginning as Development Academic Partners. Mutual 
interest is instrumental in this longstanding partnership. The role of the academic development partner is well 
defined and involves the following: 

• Identifying at least one local industry partner involved in software development activities. 

• Working with assigned focus groups to critically review current course. 

• Developing six hours of course modules to address identified gaps in a content area familiar to the 
university program and its local industry partner(s). 

• Assessing course contents through at least two delivery cycles. 

2.4 Implementation Academic Partners and Activities 

This project calls for delivery of the active learning tools, as well as dissemination of the tools and assessment 
reports. Six domestic and one international partner have supported this project including: Embry-Riddle 
Aeronautical University, Montana Tech, University of Michigan (Dearborn), Virginia State University, Fairfield 
University, Milwaukee School of Engineering, and ORT Braude College, Israel. Partners were identified through 
networking at the American Society for Engineering Education conferences, and sharing of the author’s research. 
A workshop held at the author’s institute in August 2015 led to the inclusion of five other implementation 
academic partners namely: Auburn University, East Carolina University, Kennesaw State University, Bowie 
State University, and Clarion University. Likewise a second workshop held at the author’s institute in August 
2016 led to the inclusion of nine other implementation academic partners namely: Rocky Mountain College, 
Kentucky State University, Mount Mercy University, University of Maryland, Baldwin Wallace University, 
Whitworth University, SUNY Oneonta, George Mason University, & Keynon College. The role of the academic 
implementation partner is well defined and involves the following: 

• Using all or some of the active learning tools developed by this project in at least one course through at 
least two delivery cycles, and 

• Evaluating the course(s) and assessing the instruction at several levels. 

In addition, at the request of respective faculty members, the active learning tools have been shared with 
University of Alaska Southeast, Georgia Southern University, University of South Carolina—Upstate, Indiana 
University Southeast, Rose-Hulman Institute of Technology, and Eastern Mediterranean University (Cyprus). 

2.5 Focus Groups 

To ensure project plans were adequately executed, focus groups comprised of PI and/or co-PIs and academic 
development partners and industry partners were formed based on individual’s interest and expertise in the 
projects four SV&V topic areas: requirements engineering, software reviews, configuration management, and 
software testing. Well-defined activities carried out by focus groups included the following: 

• Researching course contents, understanding industry requirements, identifying inadequacies in the 
projects SV&V topic areas. 

• Searching for available SV&V teaching materials. 

• Developing course contents to address inadequacies. 

• Assisting in developing evaluation questionnaires and instalments. 

3. Project Implementation 

In the past three years, the focus groups have followed phase-wise project plans to accomplish project objectives. 
A course enhancement plan was utilized to drive activities related to reviewing, enhancing, and modularizing. A 
course delivery plan was used to drive activities related to developing content delivery strategies. An evaluation 
and assessment plan was used to drive activities related to periodically evaluating student learning and assessing 
this project. Finally, a course dissemination plan was (is being) used to drive activities related to disseminating 
course modules and assessment reports. 
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3.1 Course Enhancement Plan 

Project objectives, target population, SV&V topics, employer-sought key skill areas, and a sound development 
methodology were used by the focus groups to develop 42 hours of active learning tools consisting of Case 
Studies, Class Exercises, and Case Study Videos. The following narrative explains the execution of the course 
enhancement plan. 

The focus groups used gap analysis to assess the inadequacies of the existing course contents in relation to the 
SV&V topic areas. Focus groups met multiple times at the author’s institution. In addition conference calls, 
phone calls and emails were used for timely communication amongst group members. They identified 
Knowledge Areas (KA), whether a single KA or a combination of KAs, and their accompanying learning 
objectives. They identified contact hours required for these KAs with reference to Curriculum Guidelines for 
Undergraduate Degree Programs in Software Engineering, a Volume of the Computing Curricula Series (2004). 
Before the actual development of active learning tools, the focus groups performed a comparison of academic 
and industry requirements. They also took into consideration the following four key skill areas sought after by 
employers and identified how they would be accomplished: 

i. Communication Skills: Students will gain experience in technical communication skills through collaborative 
learning, Case Study Videos, Class Exercises, and technical presentations. This is consistent with IEEE/ACM 
Curriculum Guideline # 8 (Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering, 
a Volume of the Computing Curricula Series, 2004). 

ii. Applied Knowledge of Methods: Students will use SV&V methods in the lecture and hands-on sessions. With 
Case Studies, Class Exercises, Case Study Videos, observing practitioners at work, and expert lecture sessions, 
students will gain the ability to translate theory to practice. This is consistent with IEEE/ACM Curriculum 
Guideline # 4 (Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering, a Volume 
of the Computing Curricula Series, 2004). 

iii. Applied Knowledge of Tools: Students will use SV&V Tools. Hands-on activities will give students experience 
in the use of SV&V tools for requirements management, configuration management, defect management, and 
automated testing. This is consistent with IEEE/ACM Curriculum Guideline # 12 (Curriculum Guidelines for 
Undergraduate Degree Programs in Software Engineering, a Volume of the Computing Curricula Series, 2004). 

iv. Research Exposure: Students will learn research skills. Students will submit research assignments and participate 
in research discussions on SV&V related topics. These skills will enable students to learn how to remain current 
with industry best practices and to make educated decisions in using them. 

The groups identified a comprehensive list of Case Studies, Class Exercises, and Case Study Videos. 

3.2 Course Enhancement Methodology’ 

An iterative development methodology depicted in Figure 1 was used to ensure the modules reflected both 
academic research and industry best practices. The content development process began with a meeting of the 
focus groups at the author’s institution. The groups drafted a list of active learning content topics and delivery 
formats. The list was reviewed by the PI and co-PIs and shared with the partners for further review. The finalized 
list was then used to guide the development process. In this methodology, an industry partner or academic 
partner led the development effort through a collaborative effort. Once the contents were ready for review, they 
were shared with focus group members and subsequently with all partners. The finalized contents were then 
transferred to a shareable media where they became available for delivery, further reviews, and dissemination. 
For ease of adaptation, each active learning tool is of 25 minutes duration, with some active learning tools 
having multiple parts delivered in multiple sessions. 
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Figure 1. Development methodology 


4. Project Results 

Several course enhancement tools were developed in this work (Acharya, Manohar, Wu, Schilling, & Ansari, 
2016) by the collaborative focus groups that brought together the best practices both in industry and in academia. 
The domain knowledge was then packaged in active learning tools and modules that encourages significant 
student engagement in their learning process. Such active learning tools created in this work are described in the 
following sections. 

4.1 Active Learning Tools 

Active learning tools engage students in their learning process and lead to excitement about learning amongst the 
students. Active learning instructional strategies include a wide range of activities that share the common 
element of—involving students in doing things and thinking about the things they are doing (Bonwell & Eison, 
1991). Active learning instructional strategies can be created and used to engage students in (a) thinking 
critically or creatively, (b) speaking with a partner, in a small group, or with the entire class, (c) expressing ideas 
through writing, (d) exploring personal attitudes and values, (e) giving and receiving feedback, and (f) reflecting 
upon the learning process (Eison, 2010). It should also be noted that active learning instructional strategies can (a) 
be completed by students either in-class or out-of-class, (b) be done by students working either as individuals or 
in group, and (c) be done either with or without the use of technology tools. This project has developed three 
categories of active learning tools as described in the following sections. 

4.1.1 Case Studies 

Case Studies serve as useful tools to teach software methods & processes. Raju and Sankar (1999) define Case 
Study education as providing students with a record of a technical and/or business issue that actually has been 
faced by managers, together with surrounding facts, opinions, and prejudices upon which management decisions 
have to depend. In this project, Case Studies are drawn from a wide range of industry SV&V practices and 
reported in detail elsewhere (Manohar et al., 2015). Students are presented industry standard documents for 
review to prepare for the tasks. This project has disseminated eleven Case Studies. Examples of some Case 
Studies are Understanding User Requirements, A Software Test Plan, and Importance of Peer Reviews. A 
complete set of case studies has been published by Acharya and Manohar (2016). 

4.1.2 Class Exercises 

Class Exercises have been designed for the class time to explicitly raise questions to invite student participation. 
These exercises may be questions to think further into the concepts for a deeper understanding, or practice using 
student knowledge with hands-on practice for problem solving. The authors have disseminated sixteen exercises 
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(Wu et al., 2016). Examples of some Class Exercises are Requirements Ambiguity, Defect Lifecycle, and IEEE 
Standards. 

4.1.3 Case Study Videos 

Videos are viewed as an effective method of presenting standard material while addressing students of different 
learning styles. In this project, the Case Study Videos provide a realistic picture for the audience to appreciate 
many SV&V processes being practiced. The video scripts were first drafted by the industry partners and 
subsequently confirmed by the testimonies shared in focus group discussions. The process has been described in 
details in a previous publication (Acharya, Manohar, & Wu, 2016). Four videos, ranging from 15 to 24 minutes, 
have been disseminated. Some Video Case Study titles are Formal Inspection Scenes and Requirement Analysis 
Scenes. 

4.2 Building a Collaborative SV&V Community 

As mentioned in Section 2, the project has brought together 5 industrial partners, 2 academic development 
partners and 7 national and international academic implementation partners. In addition, over 14 different 
academic and industrial organizations have taken advantage of the summer workshops. Thus about 40-50 people 
working full time or teaching in SV&V area have been working together and collaborating in developing and 
implementing the active learning tools and thus building a thriving SV&V professional community (Acharya, 
Manohar, Wu, Schilling, & Ansari, 2014). This is a really significant result of this project that is most likely to 
continue in future well beyond the time frame of the project. 

5. Course Delivery 

The following narrative explains the instructor kit, and delivery strategy of the enhanced SV&V course that 
deployed the active learning tools in the classroom. The course was delivered at the author’s host institution as 
well as other academic institutions and industrial organizations. The academic implementation partners have 
started implementation at various levels of adaptation in their respective institutions in their 2015 and/or 2016 
terms. The industry partners are considering taking selected tools to use in their on-the-job training for 
practitioners. 

5.1 Instructor Kit 

To ensure instructors have the resources they need to start a new course or strengthen existing courses an 
instructor kit has been created. The kit consists of: active learning tools, sample syllabus, pre-tests/post-tests and 
answer key, and sample of midterm/final exams. Each active learning tool module consists of the following 
components: 

a) Active learning tool description 

b) Instruction notes 

c) Student handout 

d) Assessment instrument 

5.2 Course Delivery’ Strategy’ 

As student learning retention and eventual application of the knowledge gained depends on knowledge retention 
activities delivered in different settings it is important to identify and incorporate delivery strategies that meet 
course and/or module learning outcomes. Researchers in both the academia and the industry have published 
literature on delivery strategies. Some of the more prominent strategies used in higher education are lecture, case 
method, discussion, active learning, cooperative learning, integrating technology and distance learning (Teaching 
Strategies, http://www.gmu.edu/resources/facstaff/part-time/strategy.html). Mishra et al. (2014) report that prior 
knowledge about software quality concepts and good programming skills can help students to achieve success in 
a software verification & validation course. Furthermore they also suggest that team work can be chosen as a 
strategy, since it facilitates students’ understanding and motivates them to study. 

With the active learning tools designed to impart practical knowledge into theoretical understanding, we 
encourage a flipped classroom model (Bonwell & Eison, 1991) in which we can maximize class time for active 
learning tools so as to engage the students for further digestion of the knowledge in the context of industry 
practices. Students are expected to be prepared outside of the classroom beforehand, with assigned textbook 
readings or reviewing of online materials. For effective delivery it is also recommended that students work in small 
teams. 


74 




jel.ccsenet.org 


Journal of Education and Learning 


Vol. 6, No. 2; 2017 


The active learning tools are built on basic knowledge and engage the students in different ways to study SV&V in 
practice. The Case Studies are explicit in this approach: each Case Study makes the point to consider issues in 
realistic practices. Instructors can present the Case Study while guiding students into further study and discussion 
of the practical issues in SV&V. The Class Exercises are designed for interaction in the classroom during group 
discussions. The instructor brings the question(s) as the moderator and guides the discussion session. The 
instructor may also use the Class Exercise to lead students into subsequent group mini projects or individual mini 
projects. The Instructor Notes component of the Class Exercise discusses these possibilities. Students are likely to 
find the Case Study Videos by their nature as multimedia, as highly engaging. These videos share real-life 
perspectives of actions and their consequences. The videos are by design in sequences of scenes. For instructional 
purposes, it is highly beneficial to “pause” the video at appropriate moments so as to engage the students in 
discussion. Suggested discussion questions accompany the videos. 

As mentioned before, the active learning tools are modularized into flexible modules. Instructors may consider 
their various needs such as curriculum design, class time, and class size to adapt the active learning tools to the 
situations in their institutions. Furthermore, though we recommend it, instructors do not need to practice the 
flipped classroom model or may do so partially. 

To summarize, the course delivery plan encourages the following: 

• Using the flipped classroom model (if applicable) 

• Have students work in small teams (2-3 students per team) 

• Delivering tools in one or multiple 25-minute sessions 

• Using pre-test and post-test instruments to tailor course delivery 

• Evaluating student learning of the module immediately after delivery 

5.2.1 Course Delivery at the Author’s Institution 

The recommended flipped classroom model is used to deliver the contents at the host institution. A 
week-by-week course delivery plan, developed by the focus groups, was used to deliver a course on SV&V in 
spring 2015. The feedback from the students and the instructor was used to revise the tools and the delivery plan. 
Table 1 depicts the revised delivery schedule being used at the author’s institute in spring 2016 delivery of the 
same course. At the author’s institute this 3-credit course meets 4 hours a week. 

5.2.2 Course Delivery at other Academic Institutions 

It is recommended that the Academic Institutions follow a similar delivery plan. However, they may choose to 
offer the enhanced course materials as a new course offering or as modules that fit into their existing course 
offerings. Partners may also supplement contents with locally developed materials. These supplemental materials 
will be part of the course repository if the PI and the partners agree that such materials add to student learning 
and teaching of such materials can be easily replicated by implementing instructors. 

5.2.3 Course Delivery at Industry Partner Facilities 

Industry partners local to the author’s institute are considering delivering specific modules to their employees at 
their facilities or at the author’s institution. During assessment periods, industry partners will deliver selected 
modules and provide assessment results. Using the provided assessment tools, both academic and industry 
implementation partners are expected to have their students and instmctor(s) complete surveys pertaining to the 
tools delivered and the delivery strategy. 

6. Project Assessment and Evaluation 

Project assessment and evaluation are important activities in this project. The following narrative explains the 
overarching project goals, followed by a description of the assessment and evaluation plan by both internal and 
external evaluators. The narrative also provides samples of actual assessments conducted by the different 
stakeholders. Various stakeholders in this project include the students (SIR II surveys and in-class feedback 
surveys), faculty (faculty course assessment reports), industrial and academic partners (surveys) and external 
evaluators (surveys and assessment reports). Each of the stakeholders have been completing work diligently. 
There is an abundance of information available at this stage to identify the strengths and weaknesses of this 
project, and then chart out the future course of action as this project moves rapidly in the dissemination and 
conclusion phase. 


75 




jel.ccsenet.org 


Journal of Education and Learning 


Vol. 6, No. 2; 2017 


6.1 Project Assessment and Evaluation Plan 

The expected project outcomes that are needed to be assessed by the project evaluators and the suggested 
evaluation tools are summarized as follows: 

• Outcome 1: Project improves knowledge and skills of students 

> Assessment tools (in-use and suggested)—outcomes assessment in class (faculty), SIR II surveys 
(students), industry training feedback (industrial partners/trainers). 

• Outcome 2: Project helps evolve SV&V community —focus groups work together in developing 
appropriate pedagogical tools for enhancing SV&V education: 

> Assessment tools (in-use and suggested)—Organization of meetings, workshops and seminars, 
e-mail based communication, surveys, Linkedln group membership, revisions, YouTube videos, new 
curriculum or revision developed and delivered at other institutions, curriculum adaptations (PI, Co-PI 
to implement, external evaluators to assess this activity). 

• Outcome 3: Project improves SV&V teaching and learning opportunities 

> Assessment tools (in-use and suggested)—expanded availability of new teaching and learning tools 
and materials—books, class notes, Case Study Videos, Case Studies, Class Exercises (PI, Co-PI to 
implement, external evaluators to assess this activity). 

The focus groups have worked with the external evaluators to develop the evaluation questionnaires and 
instruments for this plan. 


Table 1. Course delivery schedule 


Week 

SV&V Focus 

Area 

Focus Area Topic 

(self-reading lectures) 

Active Learning Tool 

(in-class activities) 

W1 

SV&V 

Fundamentals 

• Software Verification and Validation 

• Relationship of Software V&V to 
software development 

• Historical Perspective of V&V 

• Independent Verification and Validation 

• CASE STUDY - Business paper analysis (legal): Kmart Sues i2 over 

Software 

• CASE STUDY- Business paper analysis (legal): Bad Software: A 

Consumer Protection Guide 

W2 

SV&V 

Fundamentals 

• S/W Quality Software Quality Assurance 

• V&V Economic Justification 

• Quality Methods in other industries 

• Overview of Software Development 
Lifecycle Models 

• Requirements 

• CLASS EXERCISE: Famous Software Bugs 

• CLASS EXERCISE: Ambiguous Questions 

• CLASS EXERCISE: Business Requirements and Functional 
Requirements 

• VIDEO CASE STUDY - DVD- Scenes of Requirements Elicitation 

W3 

Requirements 

Management 

• Requirements 

• Requirement Management 

• Software Requirement Specification 
(SRS) 

• CLASS EXERCISE: Requirement Ambiguity 

• CASE STUDY - Understanding User Requirements 

• CASE STUDY - Requirements from a Customer Perspective - 
Ambiguity 

W4 

Requirements 

Management 

• Requirements 

• Requirement Management 

• Software Requirement Specification 
(SRS) 

• CLASS EXERCISE: SRS Sentences 

• CLASS EXERCISE: Developing SRS from Need Statement 

• CASE STUDY - Requirements from a Customer Perspective - 
Ambiguity 

W5 

Requirements 

Management 

• Requirements 

• Requirement Management 

• Software Requirement Specification 
(SRS) 

• CLASS EXERCISE: Implied and Stated Requirements 

• CASE STUDY - Requirements from a Customer Perspective - 
Ambiguity 

• CLASS EXERCISE: Developing Use Case from Needs Statement 
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W6 

Software 

• Software Verification 

• 

CLASS EXERCISE: Review a document for number of “t”s. 


Reviews 

• Verification Activities, 

• 

CASE STUDY - Peer Reviews 




• 

Perform a SRS Review of the Hospital Management System SRS 

W7 

Software 

• Formal Inspection Process 

• 

CASE STUDY - Importance of Reviews 


Reviews 

• Applying the Formal Inspection Process 

• 

VIDEO CASE STUDY - DVD- Scenes of Software Inspection 

W8 

Software 

• Applying the Formal Inspection Process 

• 

CLASS EXERCISE: Formal Code Inspection 


Reviews 

• Data Collection 

• 

VIDEO CASE STUDY - DVD- Scenes of Security Inspection 

W9 

Configuration 

• Configuration Management 

• 

CLASS EXERCISE: Using a configuration management tool 


Management 


• 

CASE STUDY - Continuous Integration (Cl) 

W10 

Configuration 

• Defect Management 

• 

CLASS EXERCISE: Understanding the Defect Management Cycle 


Management 


• 

CASE STUDY - Version Control Management 

Wll 

Software 

• Software Testing 

• 

CLASS EXERCISE: Testing Tools 


Testing 

• Testing Methods 

• 

CASE STUDY - Performance Testing/ Load Testing 

W12 

Software 

• Test Plans 

• 

CLASS EXERCISE: Test Cases for a given requirement 


Testing 

• Testing Approach 

• 

CASE STUDY - Software Test Plan 

W13 

Software 

• Requirements Refinement 

• 

CLASS EXERCISE: Writing a Test Report 


Testing 

• Test Case Development 

• 

CASE STUDY - Industry Test Case Development 

W14 

Additional 

• Software Standards 

• 

CLASS EXERCISE: Understanding IEEE Standards 


Topics 

• Predictable Software Development 





• Reliability Modeling 




6.2 Examples of Project Assessment and Evaluation from External Evaluator 

The external evaluator was provided with all the student feedback forms, industrial and academic partner 
feedback forms, and also online surveys completed by the developers and users. Based on all of these diverse 
data sources, the external evaluator submitted a Year-End Program Evaluation reports for years 2014-2015, and 
2015-2016. The specific questions that were examined in this Year-End Program Evaluation were as follows: 

1) To what degree are industry and academic partners involved and satisfied with their involvement in NSF 
TUES Project # 1245036? 

Assessment Result: The data were collected from the industry partners, using a short questionnaire that 
consisted of closed- and open-ended items. Results show a strong, on-going, frequent communication and 
collaboration between the academic and industrial partners to identify contemporary and emerging 
domain-specific issues and topics and then working together to discuss appropriate pedagogical tools to be 
developed to deliver enhanced learning experiences for the students. 

2) To what degree were best-practices in industry and research utilized during development of the instructional 
materials? 

Assessment Result: The evaluation showed that all of the Class Exercises and Case Studies followed the 
intended template, and all materials required for each class activity have been developed and will be 
provided to participating instructors for future class sections. Each activity required communication skills 
and application of course content. The topics and issues delivered in the new active learning tools were 
based on the brain-storming sessions between the academic and industrial partners to maintain a strong 
currency of knowledge. 

3) To what extent did students learn from and find the newly developed materials to be useful? 

Assessment Result: Basic psychometric analysis of the items on each pre- and post-test was conducted, and 
a paired samples t test was conducted to determine significant differences in the pre- and post-scores. The 
data show some improvement from pre- to post-test. There was a mean difference of approximately one 
point, with the post-test scores being higher. It should be noted that the sample size is small (9 students) for 
utilizing t tests and item analysis, and the Principal Investigators should use the results from these analyses 
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keeping the limitation of sample size in mind. In future semesters, similar analysis will be tun based on data 
collected from a variety of classes at several different institutions, and larger sample sizes are likely to 
provide more usable information. 

6.3 Examples of Project Assessment and Evaluation from Academic Partners 

Several academic partners adapted the newly developed curriculum to different degrees and also delivered some 
of the active learning tools. The effectiveness of the curriculum and the tools was assessed based on the data 
collected from the students, including pre- and post-tests, a general evaluation of course instructional activities, 
and detailed evaluations of specific pedagogical tools. The data sources collected from students included the 
following: 

• Pre-post Quiz Related to IEEE Standards 

• Pre-Post Quiz Related to Configuration Management 

• Pre-Post Quiz Related to Requirements, Requirements Engineering, Requirements Management 

• Pre-Post Quiz Related to Software Peer Reviews 

• Pre-Post Survey on Software V & V Focus Areas 

• Post-Only General Student Evaluation of Couse Instructional Activities 

• Post-Only Student Evaluation of Specific Course Instructional Activities 

Assessment Results: Basic psychometric analysis of the items on each pre- and post-test was conducted, 
and a paired sample t test was conducted to determine significant differences in the pre- and post-scores. It 
was found that there was significant improvement in students’ post-test scores in the areas of IEEE 
standards and configuration management. For other areas, there was some improvement, but it is determined 
to be statistically significant at this time in the data collection process. Some of the assessment questions 
may need to be rewritten to improve clarity, while for other questions, an increase in the degree of difficulty 
would help in discerning differences in student learning. 

6.4 Student Feedback 

The student feedback about specific pedagogical tools was also obtained in the form of open ended questions. 
These data are currently being analyzed quantitatively and will be reported at a later date. Some of the 
representative comments from the student feedback are summarized below: 

• Security Inspection Video: This video did facilitate small and large group discussions. I thought the 
acting in the video was actually pretty decent it portrayed events that could have actually happened. It’s 
always good to have students think about subject and then explain the right thing to them because it shows 
if they were right or not. It was nice to see examples of security vulnerability and some ways to look to 
prevent this. 1 understand to always cover security thoroughly and often during development. 

• Cost Effective Testing Exercise: Fun and informative! This was good activity to think critically about 
how to test a system. 

• Software Requirements Specification (SRS) Review Exercise: Useful. Very long and detailed. Does 
look like a real world document but if made shorter could get through entire document. 

• Test Case Development: It helped to show the importance of working through case studies. I thoroughly 
enjoyed the activity. I was able to analyze proper methods of creating test cases. Practical and straight to 
the point. I enjoyed comping up with individual solutions before discussing as a group. 

6.5 Examples of Project Assessment and Evaluation from the Instructor 

One of the authors has been delivering a SV&V course since 2005 and is required to perform an ABET Criterion 
3 outcomes assessment. In the spring 2013 term, when the new pedagogical tools were not yet available, there 
was a weakness associated with the ABET learning outcome “<?” (an ability to identify, formulate, and solve 
engineering problems), where less than 60% of the students scored better than 80% on the assessment tasks. 
When appropriate Case Studies were developed in 2014 and implemented in the Spring 2105 and 2016 terms, 
the student performance related to outcome “<?” rose to the excellent range (>=90%). This presents clear 
evidence that the Case Study based teaching method is more effective in supporting student learning and 
increasing the ability to identify, formulate, and solve engineering problems. Therefore, Case Study based 
educational tools will be progressively developed, adopted, and delivered in several other aspects of the SV&V 
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area such as legal issues in software, software consumer protection, and requirements from the customers’ 
perspectives. The results of those implementations will be reported later as more data become available. 

6.6 Students ’ Perceptions of the Course Instructional Materials 

Table 2 provides a summary of students’ perceptions of the course instructional materials. Twelve students 
completed the questionnaire. Students were asked to rate each type of assignment or activity according to the 
scales shown in Table 2. The results show that 90% or more of the students who just completed the class found 
the exercises, case studies, video case studies, full class discussions, and projects to be moderately or extremely 
useful. Students found less utility in the written homework assignments and textbook readings. 

The end of course questionnaire also asked students about how often they do several tasks in a typical course in 
their major and in the current class. Results shown in Table 3 suggest that in typical classes in their major, the 
majority of students pay attention in class, but only engage in small group and large class discussions to a small 
or moderate degree. 

When asked the same questions about the current class, results related to many items were similar, but more 
students reported that they complete real world exercises and feel accountable to classmates in full class 
discussions (Table 4). These results support the inclusion of the hands-on, real-world application activities as 
well as the integration of communication skills into the activities. 


Table 2. Descriptive results from the general evaluation of course instructional activities part 1 


Rate the value of the assignments/activities completed in this class. Address the activities as a whole, rather than focusing on one single 

instance. 

Activity 

NA/Don’t 

Know (0) 

Not Useful 

(1) 

Somewhat 

Useful 

Moderately 

Useful 

Extremely 

Useful 

Descriptive 

Stats 






(2) 


(3) 


(4) 





n % 

n 

% 

n % 

n 

% 

n 

% 

M 

SD 

i. 

Lecture 





2 

16.7 

3 

25.0 

4.4 

0.79 

2. 

Exercises 





1 

8.3 

3 

25.0 

4.6 

0.67 

3. 

Case Studies 


i 

8.3 




3 

25.0 

4.4 

1.16 

4. 

Video Case Study 





3 

25.0 

4 

33.3 

4.2 

0.83 

5. 

Small Group Discussions 





2 

16.7 

3 

25.0 

4.4 

0.79 

6. 

Full Class Discussions 





1 

8.3 

5 

41.7 

4.4 

0.67 

7. 

Written Individual 

Homework Assignments 





5 

41.7 

1 

8.3 

4.1 

1.00 

8. 

Textbook Readings 


6 

50.0 

2 16.7 

1 

8.3 

1 

8.3 

2.3 

1.60 

9. 

Presentations 




1 8.3 

4 

33.3 

2 

16.7 

3.9 

1.08 

10. 

Tests / Quizzes 





3 

25.0 

5 

41.7 

4.1 

0.79 

11. 

Projects 


4 

33.3 


1 

8.3 

2 

16.7 

3.3 

1.83 
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Table 3. Descriptive results from the general evaluation of course instructional activities part 2 
Address how often you do each of the following in a TYPICAL DAY IN A CLASS in your major: 

NA/Don’t Not at all To a small To a moderate To a large Descriptive 




Know 

(0) 

(1) 


degree 

(2) 

degree 

(3) 


degree 

(4) 

Statistics 

Activity 

n % 

n 

% 

n 

% 

n 

% 

n 

% 

M 

SD 

12. 

Pay attention during at least 90% of 
the class session 




i 

8.3 

5 

41.7 

6 

50.0 

4.4 

0.67 

13. 

Engage with classmates in small 
group discussion about course 

content 


i 

8.3 

2 

16.7 

3 

25.0 

6 

50.0 

4.2 

1.03 

14. 

Complete real-world applications of 

course content 




2 

16.7 

4 

33.3 

6 

50.0 

4.3 

0.78 

15. 

Think critically about course content 




2 

16.7 

4 

33.3 

6 

50.0 

4.3 

0.78 

16. 

Engage with classmates in full class 
discussion about course content 


2 

16.7 

4 

33.3 

2 

16.7 

4 

33.3 

3.7 

1.15 

17. 

Check texts or other communication 

via a handheld device 

1 8.3 

5 

41.7 

3 

25.0 

2 

16.7 

1 

8.3 

2.8 

1.14 

18. 

Feel accountable for my 

contribution to class. 




4 

33.3 

4 

33.3 

4 

33.3 

4.0 

0.85 

19. 

Review research in the field. 


2 

16.7 

2 

16.7 

5 

41.7 

3 

25.0 

3.8 

1.06 

20. 

Utilize/refer to professional 

standards. 


1 

8.3 



5 

41.7 

6 

50.0 

4.3 

0.89 


Table 4. Descriptive results from the general evaluation of course instructional activities part 3 


Address how often you do each of the following in a TYPICAL DAY IN THIS CLASS: 




NA/Don’t Not at 

To a small 

To a moderate 

To a large 

Descriptive 



Know 

all 

degree 


degree 

degree 

Statistics 



(0) 

(1) 


(2) 


(3) 


(4) 




Activity: 

n % n 

% 

n 

% 

n 

% 

n 

% 

M 

SD 

21. 

Pay attention during at least 90% of 
the class session 





6 

50.0 

6 

50.0 

4.5 

0.52 

22. 

Engage with classmates in small 
group discussion about course 

content 



i 

8.3 

4 

33.3 

7 

58.3 

4.5 

0.67 

23. 

Complete real-world applications 
of course content 





5 

41.7 

7 

58.3 

4.6 

0.51 

24. 

Think critically about course 





4 

33.3 

8 

66.7 

4.7 

0.49 


content 











25. 

Engage with classmates in full class 
discussion about course content 



2 

16.7 

4 

33.3 

6 

50.0 

4.3 

0.78 

26. 

Check texts or other 

communication via a handheld 

device 

1 9.1 5 

45.5 

4 

36.4 



1 

9.1 

2.5 

1.04 

27. 

Feel accountable for my 
contribution to class. 

1 

8.3 

3 

25.0 

2 

16.7 

6 

50.0 

4.1 

1.08 

28. 

Review research in the field. 

2 

16.7 

3 

25.0 

3 

25.0 

4 

33.3 

3.8 

1.14 

29. 

Utilize/refer to professional 
standards. 



2 

16.7 

3 

25.0 

7 

58.3 

4.4 

0.79 
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7. Course Dissemination 

In consonance with NSF’s broad-dissemination requirement, the project materials and assessment reports have 
been disseminated through multiple channels. The following narrative explains the execution of the course 
dissemination plan. 

7.1 SV&V Workshops 

The project team hosted two, two days Software V&V workshop. In the August 2015 workshop eleven academic 
implementation partners, including academic development partners, travelled to Pittsburgh, PA. In addition 6 
representatives from industry who had participated in the development of materials also attended. In the August 
2016 workshop nine new academic implementation partners, one academic development partners, and 1 new 
industry partner and 1 representatives from the industry participated. The objective of this workshop was to share 
the course modules and to take the participants through delivery strategies. During this workshop, 
representatives of focus groups led the participants through a subset of the materials and also explained in detail 
the delivery strategies. Half a day was allocated for each active learning tool category. It is felt that through this 
workshop a strong interaction between facilitators and peer learners was established. Both the academic and 
industrial participants were provided with a set of the larger materials on a flash drive and a link to the instructor 
kit. 

7.2 Centralized Course Repository 

In this project, initially Dropbox was used as the centralized course repository. Forty-two hours of active 
learning tools have been stored in Dropbox. Now the active learning tools are available through the project 
website www.rmu.edu/nsfvv (depicted in Figure 2) and ENSEMBLE, a Computing Portal connecting 
Computing Educators, accessible through www.computingportal.org (depicted in Figure 3). The tools and 
supporting documents are organized based upon SV&V topics. Folders are provided for activities related to 
Configuration Management, Requirements Management, Software Reviews, and Software Testing. Underneath 
each of these folders are folders for the active learning tools: Case Studies, Class Exercises, Case Study Videos, 
and Topical Assessments. For greater availability, the videos have been uploaded to YouTube. Figure 4 depicts a 
scene of a Case Study Video as seen in YouTube. 

7.3 Conferences and Publications 

Research papers on project activities and assessments have been or are in the process of being presented at 
multiple conferences. To date, conference papers have been presented at ASEE 2014, ASEE 2015, EDSIG 2015, 
ASEE 2016, and WMSCI 2016 annual conferences. A keynote address was also delivered in WMSCI 2016. In 
2016 this project was presented at the NSF Showcase at SIGCSE 2016 and in the Envisioning the Future of 
Undergraduate STEM Education: Research and Practice symposium organized by AAAS. A workshop was 
conducted in EDSIG 2016. Research work related to this project has been published in multiple journals. A book 
on SV&V Case Studies has been published by the Alexandria Street Press (online). 

7.4 Linkedln Group 

A Linkedln group called “Software V&V Community” has been established to enable SV&V implementers to 
avail themselves of peer-to-peer learning opportunities. This moderated group provides a venue to educate and to 
consult. Implementers have shared ideas and best practices, asked and answered questions, and collaborated. 
This group is open to all interested parties. 
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Figure 2. Active learning tools in project website 
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Figure 4. YouTube video presentation of the video “Security Inspection Scenes” 


8. Summary 

The lack of SV&V awareness in industry and more importantly, the shortage of SV&V practitioners poses 
multitude of problems in the software business community. Imparting real world experiences in academia as 
well as the industry is a challenge due to a lack of effective active learning tools. Through a vibrant 
academia-industry partnership and academic research, this project funded by a NSF-TUES grant has developed, 
delivered, and disseminated 42 delivery hours of active learning tools which include Case Studies, Class 
Exercises, and Case Study Videos in specific SV&V topics viz. requirements engineering, configuration 
management, software reviews, and software testing. Relevant active learning tools and appropriate supporting 
instructor’s supplementary materials have been prepared. Applicable student learning outcomes for the software 
V&V course have been determined and their relationship to ABET criteria and revised Bloom’s taxonomy for 
STEM disciplines has been mapped. The effectiveness of active learning tools has been determined based on this 
evaluation context. Project evaluation is obtained through pre- and post-quizzes, student SIR-II surveys, 
instructor’s reflections, ABET outcomes analysis, and external evaluator assessment. The results show significant 
increase in student engagement, excitement, satisfaction, and learning outcomes in all domains of SV&V 
knowledge addressed thus validating the fundamental pedagogical hypothesis of this work. The active learning 
tools developed have been disseminated through a workshop and other means to thirty-three universities and six 
industry partners to date. The information and data dissemination has also been achieved through several 
conferences presentations and journal and conference publications. Beyond the enhanced SV&V course 
pedagogy itself, this project has contributed to the building of the SV&V professional community, by bringing 
together industry and academic partners. Any software instructor or practitioner interested in gaining free access 
to the materials developed through this project is encouraged to contact the authors and expand this SV&V 
professional circle further. 
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